Apparatus and method for measuring physiological information of living subject in vehicle

ABSTRACT

Apparatus and method for measuring physiological information of a living subject in a vehicle are provided. The apparatus includes an illumination unit configured to emit light at a wavelength and illuminate a region of interest (ROI) of the living subject with the emitted light; a light sensing unit configured to remotely acquire signals of light in a range of wavelengths reflected from the ROI of the living subject responsive to the illumination; a remote photoplethysmography (PPG) extraction unit configured to extract PPG signals from the acquired signals of light; a PPG processing unit configured to process the extracted PPG signals in the form of vital signs and subsequently determine a physical and psychological status of the living subject according to the vital signs; and a control unit configured to activate the illumination unit in a time-varying pattern and to activate the light sensing unit accordingly.

FIELD OF THE INVENTION

The invention relates generally to collection of physiologicalinformation, and more particularly, to an apparatus and method formeasuring physiological information revealing one or more vital signs ofone or more living subjects from video or light signal captured of theseliving subjects inside a cabin of a vehicle, including, but not limitedto, cars, trains, ships and aircrafts.

BACKGROUND OF THE INVENTION

The background description provided herein is for the purpose ofgenerally presenting the context of the present invention. The subjectmatter discussed in the background of the invention section should notbe assumed to be prior art merely as a result of its mention in thebackground of the invention section. Similarly, a problem mentioned inthe background of the invention section or associated with the subjectmatter of the background of the invention section should not be assumedto have been previously recognized in the prior art. The subject matterin the background of the invention section merely represents differentapproaches, which in and of themselves may also be inventions.

Physiological vital signs, for example, the body temperature (BT), theheart rate (HR), and/or the respiration rate (RR), along with thequantitative features extracted from these vital signs, such as heartrate variability (HRV) that measures the variation of beat-to-beat timeinterval, are direct indicators of a person's physical and psychologicalstate and wellness. Therefore, persistent monitoring of vital signsprevails in clinic, gym and home settings for healthcare, fitness andleisure purposes.

Photoplethysmography (PPG) refers to the optical technique that measuresthe volumetric changes of an organ or body part. It utilizes the minutevariation of light absorption of certain surface or volumetric region ofinterest (ROI) during physiological activities such as heart beating orbreathing. PPG is often obtained by a pulse oximeter which records thetime-lapsed change of light reflected from or transmitted through one ormore ROI. This time-varying light absorption picked up by a pulseoximeter is a function of changes in arterial blood oxygen saturationand changes in blood volume in the skin.

Typically, a pulse oximeter comprises a red light emitting diode (LED)and an infrared LED as an illumination unit and a photodetector as alight sensing unit. Under the most common transmissive operation mode,the illumination unit contacts one side of a thin part of a livingsubject's body, such as a fingertip, earlobe, or forehead, emitting twowavelengths of light that are picked up by the light sensing unit fromthe other side. The illumination unit quickly switches between shots ofred and infrared light, allowing the photodetector to measure thechanging absorbance at each wavelength. Subsequent analysis of suchdual-wavelength measurement (time division multiplexing) reveals theabsorbance due to the arterial blood alone, removing artefacts fromvenous (deoxygenated) blood, skin, bone, muscle, fat, etc.

Although PPG is a favorable means of gauging vital signs as beingsimple, low-cost, and non-invasive, its nature of requiring directcontact between the PPG system and the living subject's body may lead tounpleasant and obtrusive experience. To ensure a good sensor attachment,most commercially available pulse oximeters are clamped onto fingertipsor earlobes, exerting a pressure over the living subject's tissue thatcould cause discomfort. In addition, the PPG system and any attachedcable would also prevent the subject from performing certain movement orwork.

Therefore, a heretofore unaddressed need exists in the art to addressthe aforementioned deficiencies and inadequacies.

SUMMARY OF THE INVENTION

The present invention relates to apparatus and method for measuringphysiological information revealing one or more vital signs of one ormore living subjects from video captured of these living subjects insidea cabin of a vehicle, including, but not limited to, cars, trains, shipsand aircrafts. To ensure system robustness under in-cabin settings, theinvention comprises an illumination unit that enables one or morereflective and time-varying signals containing physiological informationof one or more living subjects, a light sensing unit that acquires suchsignals, a physiological information extraction unit that extractsphysiological information encoded in the reflective signals, aprocessing unit that estimates physical and psychological status of theone or more living subjects from the obtained physiological information.

In one aspect of the invention, an apparatus for measuring physiologicalinformation of at least one living subject in a vehicle comprises anillumination unit configured to emit light at a wavelength andilluminate at least one ROI of the at least one living subject with theemitted light; a light sensing unit configured to remotely acquiresignals of light in a range of wavelengths reflected from the at leastone ROI of the at least one living subject responsive to theillumination; a remote PPG extraction unit configured to extract PPGsignals from the acquired signals of light; a PPG processing unitconfigured to process the extracted PPG signals in the form of vitalsigns and subsequently determine a physical and psychological status ofthe at least one living subject according to the vital signs; and acontrol unit configured to activate the illumination unit in atime-varying pattern and to activate the light sensing unit accordingly.In one embodiment, the time-varying pattern is a continuous pattern, aperiodic pattern, or a non-periodic pattern.

In one embodiment, the wavelength of the emitted light is about 950 nm,and the range of wavelengths of the reflected light is about 900-1000nm.

In one embodiment, the acquire signals of light comprise images of theat least one ROI of the at least one living subject responsive to theillumination.

In one embodiment, the vital signs comprise an HR, an RR, an HRV, and aBT.

In one embodiment, the illumination unit comprises a light source,wherein the light source comprises a laser, light-emitting diodes(LEDs), light bulbs, or a combination thereof.

In one embodiment, the sensing unit comprises a camera, a charge-coupleddevice (CCD), or a photon sensor array.

In one embodiment, the PPG processing unit comprises at least onebandpass filter with desired cutoff frequencies.

In one embodiment, the apparatus further comprises a face recognitionunit configured to detect and recognize facial expression, head pose,eye gaze of the at least one living subject for evaluating a status ofattention, fatigue and emotion of the at least one living subject.

In one embodiment, the apparatus also comprises a driving styleassessment unit configured to monitor maneuver operations and motiondata of the vehicle for evaluating performance of the at least oneliving subject who drives the vehicle.

In one embodiment, the apparatus may further comprise an in-cabinassistant unit configured to alert the at least one living subject whodrives the vehicle when the physical and psychological status of the atleast one living subject is determined to be abnormal, and/or to contactan emergency service in case of emergency.

In another aspect of the invention, a method for measuring physiologicalinformation of at least one living subject in a vehicle comprisesilluminating at least one region of interest (ROI) of the at least oneliving subject with light at a wavelength; remotely acquiring signals oflight in a range of wavelengths reflected from the at least one ROI ofthe at least one living subject responsive to the illumination;extracting PPG signals from the acquired signals of light; andprocessing the extracted PPG signals in the form of vital signs andsubsequently determining a physical and psychological status of the atleast one living subject according to the vital signs.

In one embodiment, the method further comprises detecting andrecognizing facial expression, head pose, eye gaze of the at least oneliving subject so as to evaluate a status of attention, fatigue andemotion of the at least one living subject.

In one embodiment, the method further comprises monitoring maneuveroperations and motion data of the vehicle so as to evaluate performanceof the at least one living subject who drives the vehicle.

In one embodiment, the method further comprises alerting the at leastone living subject who drives the vehicle when the physical andpsychological status of the at least one living subject is determined tobe abnormal, and/or contacting an emergency service in case ofemergency. In one embodiment, the light at the wavelength is emittedfrom a light source, wherein the light source comprises a laser, LEDs,light bulbs, or a combination thereof.

In one embodiment, the signals of light in the range of wavelengths areacquired by a camera, a CCD, or a photon sensor array.

In one embodiment, the acquire signals of light comprise images of theat least one ROI of the at least one living subject responsive to theillumination.

In one embodiment, the wavelength of the emitted light is about 950 nm,and the range of wavelengths of the reflected light is about 900-1000nm.

In yet another aspect, the invention relates to a non-transitorytangible computer-readable medium storing instructions which, whenexecuted by one or more processors, cause a method for measuringphysiological information of at least one living subject in a vehicle tobe performed. In one embodiment, the method comprises illuminating atleast one ROI of the at least one living subject with light at awavelength; remotely acquiring signals of light in a range ofwavelengths reflected from the at least one ROI of the at least oneliving subject responsive to the illumination; extracting PPG signalsfrom the acquired signals of light; and processing the extracted PPGsignals in the form of vital signs and subsequently determining aphysical and psychological status of the at least one living subjectaccording to the vital signs.

In one embodiment, the method further comprises detecting andrecognizing facial expression, head pose, eye gaze of the at least oneliving subject so as to evaluate a status of attention, fatigue andemotion of the at least one living subject.

In one embodiment, the method further comprises monitoring maneuveroperations and motion data of the vehicle so as to evaluate performanceof the at least one living subject who drives the vehicle.

In one embodiment, the method further comprises alerting the at leastone living subject who drives the vehicle when the physical andpsychological status of the at least one living subject is determined tobe abnormal, and/or contacting an emergency service in case ofemergency.

These and other aspects of the present invention will become apparentfrom the following description of the preferred embodiments, taken inconjunction with the following drawings, although variations andmodifications therein may be affected without departing from the spiritand scope of the novel concepts of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate one or more embodiments of theinvention and, together with the written description, serve to explainthe principles of the invention. The same reference numbers may be usedthroughout the drawings to refer to the same or like elements in theembodiments.

FIG. 1 shows schematically an apparatus for measuring physiologicalinformation of at least one living subject in a vehicle according to oneembodiment of the invention.

FIG. 2 shows schematically an apparatus for measuring physiologicalinformation of at least one living subject in a vehicle according to oneembodiment of the invention.

FIG. 3 shows a diagram of the absorption spectrum of oxygenated (HbO₂)and deoxygenated (Hb) hemoglobin in blood.

FIG. 4 shows a diagram of the energy spectrum of sunlight radiation atthe top of the atmosphere and at the sea level, including substances ofmajor absorption bands.

FIG. 5 shows a diagram of a PPG signal acquired according to oneembodiment of the invention.

FIG. 6 shows schematically a flowchart for measuring physiologicalinformation of at least one living subject in a vehicle according to oneembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention will now be described more fully hereinafter withreference to the accompanying drawings, in which exemplary embodimentsof the invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein. Rather, these embodiments are provided sothat this invention will be thorough and complete, and will fully conveythe scope of the invention to those skilled in the art. Like referencenumerals refer to like elements throughout.

The terms used in this specification generally have their ordinarymeanings in the art, within the context of the invention, and in thespecific context where each term is used. Certain terms that are used todescribe the invention are discussed below, or elsewhere in thespecification, to provide additional guidance to the practitionerregarding the description of the invention. For convenience, certainterms may be highlighted, for example using italics and/or quotationmarks. The use of highlighting has no influence on the scope and meaningof a term; the scope and meaning of a term is the same, in the samecontext, whether or not it is highlighted. It will be appreciated thatsame thing can be said in more than one way. Consequently, alternativelanguage and synonyms may be used for any one or more of the termsdiscussed herein, nor is any special significance to be placed uponwhether or not a term is elaborated or discussed herein. Synonyms forcertain terms are provided. A recital of one or more synonyms does notexclude the use of other synonyms. The use of examples anywhere in thisspecification including examples of any terms discussed herein isillustrative only, and in no way limits the scope and meaning of theinvention or of any exemplified term. Likewise, the invention is notlimited to various embodiments given in this specification.

It will be understood that, as used in the description herein andthroughout the claims that follow, the meaning of “a”, “an”, and “the”includes plural reference unless the context clearly dictates otherwise.Also, it will be understood that when an element is referred to as being“on” another element, it can be directly on the other element orintervening elements may be present therebetween. In contrast, when anelement is referred to as being “directly on” another element, there areno intervening elements present. As used herein, the term “and/or”includes any and all combinations of one or more of the associatedlisted items.

It will be understood that, although the terms first, second, third etc.may be used herein to describe various elements, components, regions,layers and/or sections, these elements, components, regions, layersand/or sections should not be limited by these terms. These terms areonly used to distinguish one element, component, region, layer orsection from another element, component, region, layer or section. Thus,a first element, component, region, layer or section discussed belowcould be termed a second element, component, region, layer or sectionwithout departing from the teachings of the invention.

Furthermore, relative terms, such as “lower” or “bottom” and “upper” or“top,” may be used herein to describe one element's relationship toanother element as illustrated in the figures. It will be understoodthat relative terms are intended to encompass different orientations ofthe device in addition to the orientation depicted in the figures. Forexample, if the device in one of the figures is turned over, elementsdescribed as being on the “lower” side of other elements would then beoriented on “upper” sides of the other elements. The exemplary term“lower”, can therefore, encompasses both an orientation of “lower” and“upper,” depending of the particular orientation of the figure.Similarly, if the device in one of the figures is turned over, elementsdescribed as “below” or “beneath” other elements would then be oriented“above” the other elements. The exemplary terms “below” or “beneath”can, therefore, encompass both an orientation of above and below.

It will be further understood that the terms “comprises” and/or“comprising,” or “includes” and/or “including” or “has” and/or “having”,or “carry” and/or “carrying,” or “contain” and/or “containing,” or“involve” and/or “involving, and the like are to be open-ended, i.e., tomean including but not limited to. When used in this invention, theyspecify the presence of stated features, regions, integers, steps,operations, elements, and/or components, but do not preclude thepresence or addition of one or more other features, regions, integers,steps, operations, elements, components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of the relevant art and thepresent invention, and will not be interpreted in an idealized or overlyformal sense unless expressly so defined herein.

As used herein, the phrase at least one of A, B, and C should beconstrued to mean a logical (A or B or C), using a non-exclusive logicalOR. As used herein, the term “and/or” includes any and all combinationsof one or more of the associated listed items.

As used herein, the term module may refer to, be part of, or include anApplication Specific Integrated Circuit (ASIC); an electronic circuit; acombinational logic circuit; a field programmable gate array (FPGA); aprocessor (shared, dedicated, or group) that executes code; othersuitable hardware components that provide the described functionality;or a combination of some or all of the above, such as in asystem-on-chip. The term module may include memory (shared, dedicated,or group) that stores code executed by the processor.

The terms chip or computer chip, as used herein, generally refers to ahardware electronic component, and may refer to or include a smallelectronic circuit unit, also known as an integrated circuit (IC), or acombination of electronic circuits or ICs.

As used herein, the term microcontroller unit or its acronym MCUgenerally refers to a small computer on a single IC chip that canexecute programs for controlling other devices or machines. Amicrocontroller unit contains one or more CPUs (processor cores) alongwith memory and programmable input/output (I/O) peripherals, and isusually designed for embedded applications.

The term interface, as used herein, generally refers to a communicationtool or means at a point of interaction between components forperforming wired or wireless data communication between the components.Generally, an interface may be applicable at the level of both hardwareand software, and may be uni-directional or bi-directional interface.Examples of physical hardware interface may include electricalconnectors, buses, ports, cables, terminals, and other I/O devices orcomponents. The components in communication with the interface may be,for example, multiple components or peripheral devices of a computersystem.

The term code, as used herein, may include software, firmware, and/ormicrocode, and may refer to programs, routines, functions, classes,and/or objects. Some or all code from multiple modules may be executedusing a single (shared) processor. In addition, some or all code frommultiple modules may be stored by a single (shared) memory. Further,some or all code from a single module may be executed using a group ofprocessors. Moreover, some or all code from a single module may bestored using a group of memories.

The apparatuses and methods will be described in the following detaileddescription and illustrated in the accompanying drawings by variousblocks, components, circuits, processes, algorithms, etc. (collectivelyreferred to as “elements”). These elements may be implemented usingelectronic hardware, computer software, or any combination thereof.Whether such elements are implemented as hardware or software dependsupon the particular application and design constraints imposed on theoverall system. By way of example, an element, or any portion of anelement, or any combination of elements may be implemented as a“processing system” that includes one or more processors. Examples ofprocessors include microprocessors, microcontrollers, graphicsprocessing units (GPUs), central processing units (CPUs), applicationprocessors, digital signal processors (DSPs), reduced instruction setcomputing (RISC) processors, systems on a chip (SoC), basebandprocessors, field programmable gate arrays (FPGAs), programmable logicdevices (PLDs), state machines, gated logic, discrete hardware circuits,and other suitable hardware configured to perform the variousfunctionality described throughout this disclosure. One or moreprocessors in the processing system may execute software. Software shallbe construed broadly to mean instructions, instruction sets, code, codesegments, program code, programs, subprograms, software components,applications, software applications, software packages, routines,subroutines, objects, executables, threads of execution, procedures,functions, etc., whether referred to as software, firmware, middleware,microcode, hardware description language, or otherwise.

Accordingly, in one or more example embodiments, the functions describedmay be implemented in hardware, software, or any combination thereof. Ifimplemented in software, the functions may be stored on or encoded asone or more instructions or code on a computer-readable medium.Computer-readable media includes computer storage media. Storage mediamay be any available media that can be accessed by a computer. By way ofexample, and not limitation, such computer-readable media can comprise arandom-access memory (RAM), a read-only memory (ROM), an electricallyerasable programmable ROM (EEPROM), optical disk storage, magnetic diskstorage, other magnetic storage devices, combinations of theaforementioned types of computer-readable media, or any other mediumthat can be used to store computer executable code in the form ofinstructions or data structures that can be accessed by a computer.

The description below is merely illustrative in nature and is in no wayintended to limit the invention, its application, or uses. The broadteachings of the invention can be implemented in a variety of forms.Therefore, while this invention includes particular examples, the truescope of the invention should not be so limited since othermodifications will become apparent upon a study of the drawings, thespecification, and the following claims. For purposes of clarity, thesame reference numbers will be used in the drawings to identify similarelements. It should be understood that one or more steps within a methodmay be executed in different order (or concurrently) without alteringthe principles of the invention.

Certain aspects of the invention relate to apparatus and method thatutilize a remote PPG (rPPG) system to measure and evaluate physiologicalinformation of one or more living subjects inside a cabin of a vehicle,including, but not limited to, cars, trains, ships and aircrafts, so asto determine and/or monitor a physical and psychological status of theof one or more living subjects. The term a living subject, as usedherein, generally refers to a live human being. It should be noted thatit can also refer to a live animal.

A remote PPG system is an emerging technology that provides acontact-free and unobtrusive solution to vital sign monitoring. Incontrast to the conventional PPG system which requires both theillumination unit and the sensing unit in contact with a livingsubject's ROI, a remote PPG system deploys its illumination and sensingunit at a distance. A typical remote PPG system uses LED or ambientlight as the illumination unit and a monochrome or color camera as thesensing unit. While distant light irradiation and detection of remotePPG system are desired for contact-free and unobtrusive applications,they are, however, detrimental to the system's signal-to-noise (SNR)ratio and motion robustness by further attenuating the light intensity.

Recent advancements in remote PPG research and development focused onimproving SNR ratio and system robustness under indoor applicationsettings. The feasibility of acquiring PPG remotely with ambient lightillumination and a commercial color camera was demonstrated. Spatialaveraging of pixel intensity over an ROI and bandpass filtering wasapplied and green channel signal was selected as the optimal PPG output.Motion robustness of remote PPG was improved by incorporating red, greenand blue channel signals with independent component analysis (ICA),which recovers three linearly-independent signals whose linearcombination form the RGB signals. It was discovered that differentabsorption spectra of arterial blood and bloodless skin cause thevariations to occur along a very specific vector in a normalizedRGB-space and shown that such vector can contribute to more accurateremote PPG output.

In addition to clinic diagnosing and gym fitness tracking, smartin-vehicle driver monitoring is another promising field of applicationfor remote PPG. Distraction, emotion (anger, sadness) and fatigue areamong the top causes of traffic accidents and it is imperative todevelop a driver monitoring system that continuously monitors thedriver's status and acts against any adverse situation. Vital signs(e.g., HR, RR, HRV) extracted from PPG are directly related to fatigueand emotion. Contact-free and unobtrusive features of remote PPG systemmake it ideal for driver monitoring, where the driver's normal maneuverand operation should not be hindered. What is more, cameras used as thelight sensing unit in remote PPG may also capture the driver's facialexpression, which is another powerful indicator of distraction, emotionand fatigue. This gives the remote PPG system additional functionalitiesof analyzing facial expression.

In-cabin remote PPG application has a unique setting that is vastlydifferent from the environment of above mentioned indoor applications.The absence of strong ambient light during night time requires thedeployment of a dominant in-cabin illumination source, making mostambient-light-based remote PPG setups impractical. Another particularlychallenging issue is the presence of direct or reflected sunlight.Together with the shadow of other objects, they impose substantial lightnoise to the current remote PPG system. Also, the limited space insidethe cabin restricts drastic driver movement while there could be moresmall vibrations due to bumpy conditions. As a result, the remote PPGunder indoor settings may not be applicable for in-vehicle vital signmeasurements.

Referring to FIGS. 1 and 2, a remote PPG apparatus/system for measuringin-vehicle vital signs of a driver of a vehicle is schematically shownaccording to embodiments of the invention. For the purpose of conciseillustration of the invention, the following description mainly focuseson the remote PPG apparatus for the in-vehicle vital sign measurementsfor the vehicle's driver 10 who performs normal driving tasks. It shouldbe noted that the invention is also applicable to passengers in thevehicle.

In the exemplary embodiments shown in FIGS. 1 and 2, the apparatus hasan illumination unit 110, a light sensing unit 120, a remote PPGextraction unit 130, a PPG processing unit 140, and a control unit 150.Optionally, the apparatus may have one or more of a face recognitionunit 160, a driving style assessment unit 170, and an in-cabin assistantunit 180.

Specifically, the illumination unit 110 is configured to emit light(i.e., electromagnetic radiation) 111 at a wavelength and illuminate anROI, e.g., the face 11 of the driver 10 with the emitted light 111. Theemitted light 111 is then reflected from the driver's face 11, and thereflected light 121 carries a time-varying signal 15 resembling heartrate (HR) and respiration rate (RR) of the driver 10. In certainembodiment, the illumination unit 110 comprises a light source. Thelight source comprises a laser, LEDs, light bulbs, or a combinationthereof. The laser may include a fiber laser, a diode laser, or thelike; and the LEDs may include organic LEDs and the like.

The light sensing unit 120 is configured to remotely acquire signals oflight 121 in a range of wavelengths reflected from the face 11 of thedriver 10 responsive to the illumination of the emitted light 111. Theacquire signals of light 121 are images of the face 11 of the driver 10.The light sensing unit 120 comprises a camera, a CCD, or a photon sensorarray.

According to the invention, the emitted light 111 with the wavelengthand certain intensity is adapted such that when it illuminates the face11 of the driver 10, the reflected light 121 from the face 11 of thedriver 10 is, under any weather or during any time of the day, fullydetectable by the light sensing unit 120. Preferably, the wavelength ofthe emitted light 111 should maximize the variation in reflected signal121 due to driver's physiological activities while maintaining areasonable SNR ratio. Such wavelength should also address the uniquechallenges faced by in-vehicle settings.

The absence of strong ambient light at night implies little to nonebackground or passive optical signal for the driver's ROI to reflect andfor the remote PPG system to pick up. This violates the dominantillumination requirement of most existing remote PPG systems and anactive illumination source is needed. However, illumination sourcesemitting visible light may severely disturb the driver, raising safetyconcerns. An infrared illumination source is favored since it emitsradiation imperceivable to human.

Changes of light absorption of arterial blood are the major factor thatmakes PPG technology viable. When heart beats, it pumps oxygen-richblood that has a different light absorption rate than normal arterialblood, leading to a change in the light absorbed and reflected. Theamount of such change is a strong function of the wavelength of incidentradiation and should be maximized at an optimal wavelength. FIG. 3 showsthe absorption spectrum of oxygenated (HbO₂) and deoxygenated (Hb)hemoglobin, corresponding to oxygen-rich and normal blood in artery.Most state-of-the-art remote PPG systems use yellow-green light(λ=520-590 nm) and red light (λ=635-700 nm) as they representsignificant gaps in the absorption band. Another wavelength candidatewould be infrared (IR) range (λ=800-1000 nm).

Unlike indoor remote PPG applications that use artificial illuminationsources void of sunlight, in-vehicle environments suffer stronginterference from sunlight. FIG. 4 shows that the irradiation ofsunlight spans the entire spectrum so that it will inevitably pollutethe radiation from the illumination source. During vehicle movement,light directly from the sun, reflected from surroundings and speckleshadows may together produce a periodic noise signal so similar to thePPG signal that such noise cannot be removed or reduced by the remotePPG system, causing system failure. Since it is trivial to shieldsunlight under in-vehicle settings due to safety, scenic and aestheticreasons, an operation wavelength in which the sunlight has lower powerdensity is preferred. While radiation spectrum on top of the atmosphereis a smooth and continuous curve, the spectrum at the sea level exhibitsseveral trenches because light with certain wavelengths is absorbed bysubstances such as ozone (O₃), oxygen (O₂), water (H₂O), and carbondioxide (CO₂). Spectral irradiance of sunlight at these trenches is atleast 50% less than that outside of the trenches, suggesting a lowerinterference level to the remote PPG system. Trenches at about 750 nmand about 900-1000 nm are close to the visible light spectrum and thelatter is preferred for its broad bandwidth. Consequently, an opticalbandpass filter with a pass band around the 900-1000 nm trench willremove most of the sunlight interference from the system.

Based on the above discussion, in one exemplary embodiment, infraredLEDs emitting infrared light centered at about 950 nm are utilized asthe illumination unit 110, and a camera receiving light of approximately900-1000 nm with a resolution of 1280×800 is utilized as the lightsensing unit 120. The camera captures frames at about 53 frames persecond (fps).

Other embodiments of the illumination unit and the sensing unitoperating around the similar wavelengths can also be utilized topractice the invention. That is, the illumination unit emits lighthaving a wavelength centered around 950 nm, and the light sensing unitacquires the reflected light in a wavelength range of about 900-1000 nm,which in certain embodiments, can be achieved by an optical bandpassfilter with a pass band about 900-1000 nm.

Operably, the illumination unit 110 and the light sensing unit 120 areplaced within an optimal range to the driver. A distance shorter thanthis optimal range reduces the area that the light sensing unit 120 canmonitor. The driver may move out of the camera frame and the systemloses input data. A distance longer than the optimal range furtherattenuates the radiation signal, reducing SNR ratio and systemrobustness. Also, it is preferred to illuminate and capture the driver'sfrontal face to reduce shadow and increase reflection received.

Referring back to FIGS. 1 and 2, particularly to FIG. 2, the controlunit 150 is configured to activate the illumination unit 110 in atime-varying pattern and to activate the light sensing unit 120accordingly. When the illumination unit 110 is activated, it emits thelight 111, otherwise, it is turned off and emits no light. Similarly,when the light sensing unit 120 is activated, it acquires the reflectedlight 121, otherwise, it is turned off. The time-varying pattern is acontinuous pattern, a periodic pattern, or a non-periodic pattern.

Although the selected infrared light is imperceivable to human andcarries a lower energy than visible light does, extended exposure couldstill be hazardous to the driver. Thus, the control unit 150 isconfigured to turn on the illumination unit 110 only upon datacollection. Depending on the sampling rate, the illumination unit 110may be activated only 10% to 50% of the time. This control unit 150 isalso configured to synchronize and trigger the light sensing unit 120while the illumination unit 110 is activated, saving valuable memorystorage and computation power. The control unit 150 may have its ownmemory cache and computation module such as a microcontroller unit(MCU), or may share them with one or more other units.

The control unit 150 may also include other hardware and software modulefor non-uniform illumination correction and motion artefacts removal sothat subsequent PPG extraction can be performed consistently.

The remote PPG extraction unit 130 is configured to extract PPG signalsfrom the acquired signals of light, i.e., the images of the face (ROI)11 of the driver 10 acquired by the light sensing unit 120. The rawimage signal first goes through preprocessing to have motion artefactsremoved and nonuniform illumination corrected, isolating the PPG signal.The preprocessing phase may also include a face detection algorithm thatlocates the living subject's face and discard the undesired parts of theimage. For multi-subject applications, a subject segmentation unit canbe added before the PPG extraction unit to recognize and separatemultiple subjects and then computes the PPG signal for each individual.

Then one or more ROIs of width w_(i) and height h_(i) at time t_(j)

[ROI ₁(h ₁ ×w ₁ ,t _(j)), ROI ₂(h ₂ ×w ₂ ,t _(j)), . . . , ROI _(k)(h_(k) ×w _(k) ,t _(j))]

is selected from the preprocessed image and a PPG signal is computed foreach ROI by taking the spatial average over the corresponding region.

${PPG}_{i,t_{j}} = {\frac{1}{w_{i}h_{i}}{\sum_{m = 1}^{h_{i}}{\sum_{n = 1}^{w_{i}}{{ROI}_{i}( {m,n,t_{j}} )}}}}$

The time-varying remote PPG signal of an ROI is then acquired byconstructing a tuple that combines the averaged PPG signal of that ROIat various times.

PPG _(i)(t)=[PPG _(i,t) ₁ ,PPG _(i,t) ₂ , . . . , PPG _(i,t) ₃ ]

The remote PPG extraction unit may also include any hardware or softwaremodule that computes the signal quality from different ROI and outputsthe signal with the highest confidence.

For multiple living subject applications, a subject segmentation unitcan be added before the PPG extraction unit to recognize and separatemultiple subjects and then computes the PPG signal for each individual.

The PPG processing unit 140 is configured to process the extracted PPGsignals in the form of vital signs and subsequently determine a physicaland psychological status of the at least one living subject according tothe vital signs including, but not limited to an HR, an RR, an HRV and aBT.

The remote PPG signal is a superposition of different physiologicalsignals including HR and RR and these vital signs can be separated byapplying proper frequency filter. In certain embodiments, the PPGprocessing unit 140 may include at least one bandpass filter withdesired cutoff frequencies.

In one embodiment, the HR is extracted by applying a bandpass filterwith certain cutoff frequencies, corresponding to a heart rate range of30 to 300 beats per minute (bpm). FIG. 5 shows an exemplary remote PPGsignal according to one embodiment of the invention and the referencingelectrocardiography (ECG). The error for the HR has an average of 3.97bpm and a standard deviation of 1.44 bpm.

In one embodiment, the RR is extracted by applying a bandpass filterwith certain cutoff frequencies, corresponding to a respiration raterange of 6 to 24 counts per minute (cpm).

In one embodiment, HRV is computed from the HR, tracks the instantaneousheart rate based on the time interval between each heartbeat. Generally,frequency components of the HRV are classified into very low frequency(VLF, <0.04 Hz), low frequency (LF, 0.04 to 0.15 Hz) and high frequency(HF, 0.15 to 0.4 Hz). They provide insights on a living subject'sphysical and psychological status. LF/HF ratio decreases as fatiguebuilds up in during extended monotonous driving.

When only rPPG-related physiological information is present, a driver isconsidered to be alert at the beginning of a drive and the signalacquired at this time is treated as alert baseline. The inventioncontinuously measures rPPG-related physiological information at compareit with the alert baseline. Once a critical deviation from the alertbaseline is detected the driver is considered as fatigue. In certainembodiment, a multi-input fatigue determination algorithm is used ifother inputs such as facial expression and driving behavior are alsopresent.

Still referring to FIG. 2, the apparatus further has a face recognitionunit 160 configured to detect and recognize facial expression, headpose, eye gaze of the driver 10 for evaluating a status of attention,fatigue and emotion of the driver 10. The face recognition unit 160 mayinclude, but not limited to, an imaging means such as camera foracquiring images of the driver's face 11, and one or more imageprocessors for analyzing the acquired images recognize facialexpression, head pose, eye gaze, etc. of the driver 10.

The face recognition unit 160 disclosed herein is different from theaforementioned face detection algorithm that only locates the presenceof the driver's face and isolates the face region for subsequent remotePPG extraction. The face recognition unit 160, on the other hand,detects and recognizes the driver's facial expression as well as headpose and eye gaze, which would contribute to evaluating the driver'sattention, fatigue and emotion status.

Attention status refers to the driver's visual attention on trafficconditions and surrounding objects. It is evaluated using head pose andeye gaze. The face recognition unit 160 contains codes that computes thedriver's head pose and eye gaze and then determines whether the driveris paying enough visual attention. The in-cabin assistant unit 180 willalert the driver if he/she is distracted for an extended period.

Fatigue status refers to the driver's level of drowsiness and sleepinessthat hinders his/her ability to perceive and react in a proper andtimely fashion. Fatigue is usually classified, by severity, into alertphase, transitional phase, transitional to post-transitional phase andpost-transitional phase. Drivers under fatigue exhibit unique facialexpressions such as yawning, head nodding and longer blink time. Thesefatigue indicators provide another source of input for fatigue detectionin additional to the remote PPG signal.

Emotion status refers to any conscious experience and mood within adriver. Seven basic emotions include anger, contempt, fear, disgust,happiness, sadness and surprise. Intense emotions act as a source ofdistraction and could hurt one's driving ability. The face recognitionunit 160 contains codes that detects the driver's intense emotion fromfacial expression and urges the driver to calm down.

In addition, the apparatus may also have the driving style assessmentunit 170 configured to monitor maneuver operations and motion data ofthe vehicle for evaluating performance of the driver 10. The maneuveroperations may include, but not limited to, an amount of throttle, brakeand steering applied. The motion data include, but not limited to,speed, linear acceleration and angular acceleration.

Deterioration of the driver's attention, fatigue and emotion statusultimately lead to a decline of driving performance and raise safetyconcerns. For example, a tired driver would fail to react to an obstaclewithin optimal time and must perform more drastic maneuvers such assteering and braking. Thus, driving performance can be assessed bymonitoring the driver's maneuver operations and the vehicle's motion. Incertain embodiments, the driving style assessment unit 170 may includeone or more of gyro sensors, acceleration sensors and motion sensors todetect information of the maneuver operations and motion data of thevehicle. The information detected in the driving style assessment unit170 is used to extract driving style features at the beginning period ofa drive and continuously monitor the progression of driving style duringthe drive. The driver is considered as drowsy if his/her current drivingstyle deviates greatly from the initial driving style.

Driver status analysis from both face recognition unit and driving styleassessment unit should be used along with vital signs obtain by theinvention to determine the driver's physical and psychological statusprecisely.

Furthermore, the apparatus further has an in-cabin assistant unit 180configured to alert the driver when the physical and psychologicalstatus of the driver is determined to be abnormal, and/or to contact anemergency service in case of emergency. Once an absence of attention ordrowsiness is detected, the in-cabin assistant unit 180 alerts thedriver in visual, verbal or haptic means such as flashing cabin light,voice assistance or seat vibrations. The in-cabin assistant unit 180should also contact emergency service in urgent scenarios. In certainembodiments, the in-cabin assistant unit 180 may include one or morewireless transceiver communicating with a radio of the vehicle throughthe Bluetooth® in the vehicle for sending the alert to the radio whenneeded, or transmitting SOS signals to the emergency service in case ofemergency.

In the exemplary embodiments shown in FIGS. 1 and 2, the illuminationunit 110 and the light sensing unit 120 are designed as two individualcomponents. In other embodiments, the illumination unit 110 and thelight sensing unit 120 can be assembled as an integrated unit havingillumination and sensing modules. In addition, the remote PPG extractionunit, the PPG processing unit and the control unit can also assembled asan integrated unit having PPG extraction, PPG processing and controlmodules.

In addition, the apparatus also has a power module either powered withbatteries or connected to a power outlet of the vehicle for providingthe power for the above disclosed units.

In another aspect of the invention, the method for measuringphysiological information of at least one living subject in a vehicle,as shown in FIG. 6, includes the following steps.

At step S110, an ROI of the at least one living subject is illuminatedwith light at a wavelength. In one embodiment, the wavelength of thelight is about 950 nm. In one embodiment, the illuminating step S110 isperformed with the illumination unit 110 as disclosed above.

At step S120, signals of light in a range of wavelengths reflected fromthe at least one ROI of the at least one living subject responsive tothe illumination are remotely acquired. In one embodiment, the acquiredsignals of light are images of the ROI of the at least one livingsubject. In one embodiment, the range of wavelengths of the reflectedlight is about 900-1000 nm. In one embodiment, the acquiring step S120is performed with the light sensing unit 120 as disclosed above.

In one embodiment, the illuminating step 110 and subsequently theacquiring step 120 are performed in a time-varying pattern. Thetime-varying pattern is a continuous pattern, a periodic pattern, or anon-periodic pattern.

At step S130, PPG signals from the acquired signals of light areextracted. In one embodiment, the extracting step S130 is performed withthe remote PPG extraction unit 120 as disclosed above.

At step S140, the extracted PPG signals is processed in the form ofvital signs and subsequently a physical and psychological status of theat least one living subject is determined according to the vital signs.In one embodiment, the processing and determining step S140 is performedwith the PPG processing unit 140 as disclosed above.

In certain embodiments, steps S130 and S140 are respectively performedin the remote PPG extraction unit and the PPG processing unit of theabove-disclosed system. In other embodiments, steps S130 and S140 may beperformed in a server and/or cloud computing over the internet.

In addition, the method further comprises detecting and recognizingfacial expression, head pose, eye gaze of the at least one livingsubject so as to evaluate a status of attention, fatigue and emotion ofthe at least one living subject.

Furthermore, the method may include monitoring maneuver operations andmotion data of the vehicle so as to evaluate performance of the at leastone living subject who drives the vehicle, where the maneuver operationsinclude an amount of throttle, brake and steering applied, and whereinthe motion data include speed, linear acceleration and angularacceleration.

Moreover, the method also includes alerting the at least one livingsubject who drives the vehicle when the physical and psychologicalstatus of the at least one living subject is determined to be abnormal,and/or contacting an emergency service in case of emergency.

It should be noted that all or a part of the steps according to theembodiments of the invention is implemented by hardware or a programinstructing relevant hardware.

Yet another aspect of the invention provides a non-transitory tangiblecomputer-readable medium storing instructions which, when executed byone or more processors, cause the above disclosed method for measuringphysiological information of at least one living subject in a vehicle tobe performed. The computer executable instructions or program codesenable the above disclosed apparatus or a similar system to completevarious operations in accordance with the above disclosed method. Thestorage medium/memory may include, but is not limited to, high-speedrandom access medium/memory such as DRAM, SRAM, DDR RAM or other randomaccess solid state memory devices, and non-volatile memory such as oneor more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid state storage devices.

The foregoing description of the exemplary embodiments of the inventionhas been presented only for the purposes of illustration and descriptionand is not intended to be exhaustive or to limit the invention to theprecise forms disclosed. Many modifications and variations are possiblein light of the above teaching.

The embodiments were chosen and described in order to explain theprinciples of the invention and their practical application so as toenable others skilled in the art to utilize the invention and variousembodiments and with various modifications as are suited to theparticular use contemplated. Alternative embodiments will becomeapparent to those skilled in the art to which the invention pertainswithout departing from its spirit and scope. Accordingly, the scope ofthe invention is defined by the appended claims rather than theforegoing description and the exemplary embodiments described therein.

Some references, which may include patents, patent applications andvarious publications, are cited and discussed in the description of thisdisclosure. The citation and/or discussion of such references isprovided merely to clarify the description of the present disclosure andis not an admission that any such reference is “prior art” to thedisclosure described herein. All references cited and discussed in thisspecification are incorporated herein by reference in their entiretiesand to the same extent as if each reference was individuallyincorporated by reference.

1. An apparatus for measuring physiological information of at least oneliving subject in a vehicle, comprising: an illumination unit configuredto emit light at a wavelength and illuminate at least one region ofinterest (ROI) of the at least one living subject with the emittedlight; a light sensing unit configured to remotely acquire signals oflight in a range of wavelengths reflected from the at least one ROI ofthe at least one living subject responsive to the illumination; a remotephotoplethysmography (rPPG) extraction unit configured to extract PPGsignals from the acquired signals of light; a rPPG processing unitconfigured to process the extracted PPG signals in the form of vitalsigns and subsequently determine a physical and psychological status ofthe at least one living subject according to the vital signs, whereinthe rPPG processing unit comprises at least one bandpass filter withdesired cutoff frequencies; and the vital signs comprise at least aheart rate (HR), a respiration rate (RR), and a heart rate variability(HRV); wherein the RR is obtained by applying one of the at least onebandpass filter with cutoff frequencies corresponding to a respirationrate range to the extracted PPG signals; the HR is obtained by applyinganother of the at least one bandpass filter with cutoff frequenciescorresponding to a heart rate range to the extracted PPG signals, andthe HRV is computed from the HR; and a control unit configured toactivate the illumination unit in a time-varying pattern and to activatethe light sensing unit accordingly,. wherein the apparatus is configuredto assign the PPG signals including the RR, the HR and the HRV processedfrom the signals acquired at the beginning of a drive from the at leastone living subject who drives the vehicle as an alert baseline, remotelyand continuously acquire the signals from the at least one livingsubject when driving, process the acquired signals into the vital signform of the PPG signals, and compare the processed PPG signals with thealert baseline, and alert the at least one living subject as fatiguewhen a critical deviation from the alert baseline is detected.
 2. Theapparatus of claim 1, further comprising: a face recognition unitconfigured to detect and recognize facial expression, head pose, eyegaze of the at least one living subject for evaluating a status ofattention, fatigue and emotion of the at least one living subject. 3.The apparatus of claim 1, further comprising: a driving style assessmentunit configured to monitor maneuver operations and motion data of thevehicle for evaluating performance of the at least one living subjectwho drives the vehicle.
 4. The apparatus of claim 1, further comprising:an in-cabin assistant unit configured to alert the at least one livingsubject who drives the vehicle when the physical and psychologicalstatus of the at least one living subject is determined to be abnormal,and/or to contact an emergency service in case of emergency.
 5. Theapparatus of claim 1, wherein the illumination unit comprises a lightsource, wherein the light source comprises a laser, light-emittingdiodes (LEDs), light bulbs, or a combination thereof.
 6. The apparatusof claim 1, wherein the sensing unit comprises a camera, acharge-coupled device (CCD), or a photon sensor array.
 7. (canceled) 8.(canceled)
 9. The apparatus of claim 1, wherein the acquired signals oflight comprise images of the at least one ROI of the at least one livingsubject responsive to the illumination.
 10. The apparatus of claim 1,wherein the wavelength of the emitted light is about 950 nm, wherein therange of wavelengths of the reflected light is about 900-1000 nm. 11.(canceled)
 12. A method for measuring physiological information of atleast one living subject in a vehicle, comprising: illuminating at leastone region of interest (ROI) of the at least one living subject withlight at a wavelength; remotely acquiring signals of light in a range ofwavelengths reflected from the at least one ROI of the at least oneliving subject responsive to the illumination; extractingphotoplethysmography (PPG) signals from the acquired signals of light;and processing the extracted PPG signals in the form of vital signs andsubsequently determining a physical and psychological status of the atleast one living subject according to the vital signs, wherein the vitalsigns comprise at least a heart rate (HR), a respiration rate (RR), anda heart rate variability (HRV); wherein the processing step comprisesapplying one bandpass filter with cutoff frequencies corresponding to arespiration rate range to the extracted PPG signals to obtain the RR;applying another bandpass filter with cutoff frequencies correspondingto a heart rate range to the extracted PPG signals to obtain the HR, andcomputing the HRV from the HR; assigning the PPG signals including theRR, the HR and the HRV processed from the signals acquired at thebeginning of a drive from the at least one living subject who drives thevehicle as an alert baseline; remotely and continuously acquiring thesignals from the at least one living subject when driving, andprocessing the acquired signals into the vital sign form of the PPGsignals and comparing the processed PPG signals with the alert baseline,and alerting the at least one living subject as fatigue when a criticaldeviation from the alert baseline is detected.
 13. The method of claim12, further comprising: detecting and recognizing facial expression,head pose, eye gaze of the at least one living subject so as to evaluatea status of attention, fatigue and emotion of the at least one livingsubject.
 14. The method of claim 12, further comprising: monitoringmaneuver operations and motion data of the vehicle so as to evaluateperformance of the at least one living subject who drives the vehicle.15. The method of claim 12, further comprising: alerting the at leastone living subject who drives the vehicle when the physical andpsychological status of the at least one living subject is determined tobe abnormal, and/or contacting an emergency service in case ofemergency.
 16. (canceled)
 17. (canceled)
 18. The method of claim 12,wherein the acquire signals of light comprise images of the at least oneROI of the at least one living subject responsive to the illumination.19. The method of claim 12, wherein the wavelength of the emitted lightis about 950 nm, wherein the range of wavelengths of the reflected lightis about 900-1000 nm.
 20. A non-transitory tangible computer-readablemedium storing instructions which, when executed by one or moreprocessors, cause a method for measuring physiological information of atleast one living subject in a vehicle to be performed, the methodcomprising: illuminating at least one region of interest (ROI) of the atleast one living subject with light at a wavelength; remotely acquiringsignals of light in a range of wavelengths reflected from the at leastone ROI of the at least one living subject responsive to theillumination; extracting photoplethysmography (PPG) signals from theacquired signals of light; and processing the extracted PPG signals inthe form of vital signs and subsequently determining a physical andpsychological status of the at least one living subject according to thevital signs, wherein the vital signs comprise at least a heart rate(HR), a respiration rate (RR), and a heart rate variability (HRV);wherein the processing step comprises applying one bandpass filter withcutoff frequencies corresponding to a respiration rate range to theextracted PPG signals to obtain the RR; applying another bandpass filterwith cutoff frequencies corresponding to a heart rate range to theextracted PPG signals to obtain the HR, and computing the HRV from theHR; assigning the PPG signals including the RR, the HR and the HRVprocessed from the signals acquired at the beginning of a drive from theat least one living subject who drives the vehicle as an alert baseline;remotely and continuously acquiring the signals from the at least oneliving subject when driving, and processing the acquired signals intothe vital sign form of the PPG signals and comparing the processed PPGsignals with the alert baseline, and alerting the at least one livingsubject as fatigue when a critical deviation from the alert baseline isdetected.
 21. The non-transitory tangible computer-readable medium ofclaim 20, wherein the method further comprises: detecting andrecognizing facial expression, head pose, eye gaze of the at least oneliving subject so as to evaluate a status of attention, fatigue andemotion of the at least one living subject.
 22. The non-transitorytangible computer-readable medium of claim 20, wherein the methodfurther comprises: monitoring maneuver operations and motion data of thevehicle so as to evaluate performance of the at least one living subjectwho drives the vehicle.
 23. The non-transitory tangiblecomputer-readable medium of claim 20, wherein the method furthercomprises: alerting the at least one living subject who drives thevehicle when the physical and psychological status of the at least oneliving subject is determined to be abnormal, and/or contacting anemergency service in case of emergency.
 24. The apparatus of claim 1,wherein the respiration rate range is from about 6 to about 24 countsper minute (cpm); and the heart rate range is from about 30 to about 300beats per minute (bpm).
 25. The apparatus of claim 1, wherein the HRVhas frequency components classified into very low frequency (VLF) ofless than 0.04 Hz, low frequency (LF) of 0.04 Hz to 0.15 Hz, and highfrequency (HF) of 0.15 Hz to 0.4 Hz, wherein LF/HF ratio decreases asfatigue builds up in during extended monotonous driving.